Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case of deep linear neural networks. Despite the linearity of their input-output map, such networks have nonlinear gradient descent dynamics on weights that change with the addition of each new hidden layer. We show that deep linear networks exhibit nonlinear learning phenomena similar to those seen in simulations of nonlinear networks, including long plateaus followed by rapid transitions to lower error solutions, and faster convergence from greedy unsupervised pretraining initial conditions than from random initial conditions. We provide an analytical description of these phenomena by finding new exact solutions to the nonlinear dynamics of deep learning. Our theoretical analysis also reveals the surprising finding that as the depth of a network approaches infinity, learning speed can nevertheless remain finite: for a special class of initial conditions on the weights, very deep networks incur only a finite, depth independent, delay in learning speed relative to shallow networks. We show that, under certain conditions on the training data, unsupervised pretraining can find this special class of initial conditions, while scaled random Gaussian initializations cannot. We further exhibit a new class of random orthogonal initial conditions on weights that, like unsupervised pre-training, enjoys depth independent learning times. We further show that these initial conditions also lead to faithful propagation of gradients even in deep nonlinear networks, as long as they operate in a special regime known as the edge of chaos.Comment: Submission to ICLR2014. Revised based on reviewer feedbac

A mathematical theory of semantic development in deep neural networks

An extensive body of empirical research has revealed remarkable regularities in the acquisition, organization, deployment, and neural representation of human semantic knowledge, thereby raising a fundamental conceptual question: what are the theoretical principles governing the ability of neural networks to acquire, organize, and deploy abstract knowledge by integrating across many individual experiences? We address this question by mathematically analyzing the nonlinear dynamics of learning in deep linear networks. We find exact solutions to this learning dynamics that yield a conceptual explanation for the prevalence of many disparate phenomena in semantic cognition, including the hierarchical differentiation of concepts through rapid developmental transitions, the ubiquity of semantic illusions between such transitions, the emergence of item typicality and category coherence as factors controlling the speed of semantic processing, changing patterns of inductive projection over development, and the conservation of semantic similarity in neural representations across species. Thus, surprisingly, our simple neural model qualitatively recapitulates many diverse regularities underlying semantic development, while providing analytic insight into how the statistical structure of an environment can interact with nonlinear deep learning dynamics to give rise to these regularities

Observations on Eimeria from Ambystoma tigrinum, with Descriptions of Four New Species

There appear to be only three records of coccidia of the genus Ezmeria from North American Amphibia. Rankin (1937) reported Eimeria ranarum from Ambystoma opacum. Walton (personal communication) says that his record (1942) of Eimeria from Ambystoma is based upon a letter from Dr. Harold Kirby who stated that he had seen E. ranarum from A. opacum. Doran (1953) reported Eimeria grobbeni from Taricha torosa. The author, on several occasions, has noted the presence of coccidian oocysts in the rectal contents of salamanders. Four species of Eimeria have been recognized in material from Ambystoma tigrinum (Green) and will be described as new

Mapping Jewish Education: The National Picture

Based on interviews as well as a database of Jewish educational organizations, foundations, and programs, examines their accomplishments, challenges, future directions, and links within a Jewish educational system. Highlights the role of foundations

Trimitus parvus Grasse (Protozoa, Mastigophora) from a garter snake, Thamnophis radix

Reptiles collected in the vicinity of the Iowa Lakeside Laboratory, Dickinson County, Iowa, during the summer of 1950, were examined for helminths by Burnham (1951) and for protozoa by Schmidt (1954). lvlonocercomonas colubrorum, Trichomonas batrachorum and Haemogregarina sp. were found in both Thamnophis radix and T. ordinatus parietalis. An enteric flagellate of the genus Trimitus was found in a single specimen of T. radix. Special consideration is herein given to Trimitus because though this genus has been reported from North American amphibians and reptiles it appears that neither descriptions nor figures have been published, and also, because there is some uncertainty concerning the characterization of the genus

Caring for home-based care workers

Home-based care has emerged as a service delivery model to cope with the devastation caused by the HIV/AIDS epidemic in sub-Saharan Africa, where medical and traditional care infrastructures have been overwhelmed. In these communities homebased care workers provide critical services, which include physical, psychosocial, and palliative care activities.1 A quantitative and qualitative study of home-based care workers in South Africa was conducted in 2005 to better understand the needs, fears and motivations of front-line care workers at Thembalethu Home Based Care (THBC), located within the Nkomazi region of South Africa\'s Mpumalanga province. The objectives of this study were to: ■ Describe the socio-demographic background of home-based care workers to letter understand worker demographics, workers\' finances and job characteristics ■ Assess THBC care workers\' willingness to undergo voluntary counselling and testing (VCT) to determine their HIV status ■ Explore the emotional impacts of care work for THBC frontline care workers to determine what mechanisms could be put in place in order to support and expand the current care work infrastructure. Findings suggest that THBC care workers value the emotional support from weekly group meetings and use this time to process the emotional impacts of their care work. Although rates of testing are low, 83% of participants would consider undergoing VCT to learn their HIV status. Specific strategies to ensure that care workers receive appropriate medical care and supportive services are discussed. Southern African Journal of HIV Medicine Vol. 7 (3) 2006: pp. 38-4